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ABSTRACT 



A method for producing a results list for a search query 
includes producing a first results list of entries from a search 
algorithm, the first results list corresponding to a term in the 
search query, the first results list being ordered according to 
relevance rankings, modifying a feature of the first results 
lists by an editor, the modification being made according to 
an editorial rule, determining a reliability score for the 
editor, the reliability score being based upon the modifica- 
tion made the editor, combining the modification made by 
the editor in a combined search index, the combined modi- 
fication being included in the combined search index accord- 
ing to the determined reliability of the editor, and using the 
combined search index to produce a second results list 
which corresponds to the term in the search query, the 
second results list being ordered according to relevance 
rankings. 

22 Claims, 7 Drawing Sheets 
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SEARCH RESULTS USING EDITOR 
FEEDBACK 

TECHNICAL FIELD 

This invention relates to using editorial feedback of 
various Internet objects. 

BACKGROUND 

The Internet can be viewed as a large collection of 
documents, for example, text files, web-pages, newsgroup 
postings or pictures. Internet search engines provide a means 
of searching through the vast amount of documents to 
produce a results list of the documents found which match 
the terms in a search query. Typically the results list is 
presented as a list of document summaries that includes 
hyperlinks ("links") that connect each entry to the appro- 
priate Internet document. The results list is generally ranked 
by relevance (in relation to the query), with each entry 
included in the list presented either higher or lower on the 
list according to the relevance ranking as determined by the 
search engine being used. The way in which these relevance 
rankings are determined is constantly evolving as the Inter- 
net continues to evolve. 

Search engines apply different algorithms to "filter" the 
available documents and assign relevance rankings to the 
documents reviewed. The relevance rankings are generally 
stored in a search index which corresponds to documents for 
a specific search term (or related search terms). 

Initially, Internet search engines applied "content-based" 
filtering which is simply examining the number of limes that 
a query search term appeared within a document, such that 
the greater the number of times that a search term appeared, 
the more relevant the document was considered and the 
higher it was ranked. However, content-based filtering pro- 
duces rankings which are easily manipulated by the author 
of the documents reviewed, that is, an author can fill their 
web-page with multiple copies of the words that they believe 
will be searched upon and thereby increase the alleged 
relevance of their web-page. 

More advanced ranking methods apply "link analysis" 
algorithms, i.e., examining the links contained within a 
document to other documents with relatively high relevance 
rankings. However, as with the manipulation of terms in a 
web-page, as discussed above, an author can increase the 45 
number of links to other documents with high relevance 
rankings in order to increase the alleged relevance of their 
web-page. 

Editor-controlled search engines use a "siafT of editors 
(paid or volunteer) to manually select and rank individual 50 
web-page documents contained in a results list for a specific 
search term from a ranked search index. Documents may 
have their rankings changed, or document may be added and 
removed from the index over time as the editors perform 
their reviews. Since there billions of web-page documents 55 
available on the Internet, and the number of documents 
continues to grow at a tremendous pace, the amount of labor 
needed to maintain a current and complete editor-controlled 
search index is very high. The Open Directory Project 
(ODP) is an example of a co-operative editing process that 60 
uses a large number of volunteer editors to assess and 
modify the relevance rankings of documents related to a 
search term or within a specific category. ODP applies rules 
to the editor selection process. The end result of ODP is a 
editor-controlled ranking index that can be searched directly, 65 
however, the editorial feedback is not used to improve the 
efficiency of an automated search engine algorithm. 
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According to an aspect of this invention a method for 
producing a results list for a search query includes producing 
a first results list of entries from a search algorithm, the first 
results list corresponding to a term in the search query, the 
first results list being ordered according to relevance 
rankings, modifying a feature of the first results lists by an 
editor, the modification being made according to an editorial 
rule, determining a reliability score for the editor, the 
reliability score being based upon the modification made the 
editor, combining the modification made by the editor in a 
combined search index, the combined modification being 
included in the combined search index according to the 
determined reliability of the editor, and using the combined 
search index to produce a second results list which corre- 
sponds to the term in the search query, the second results list 
being ordered according to relevance rankings. 

One or more of the following features may also be 
included: producing a first results list that includes produc- 
ing a first results list that corresponds to at least one search 
term, modifying a feature according to an editorial rule that 
includes modifying any feature of the first results list, 
modifying a feature according to an editorial rule that 
includes modifying an allowed changeable feature, the 
allowed changeable feature being one or more of relevance 
ranking, entry addition, entry deletion and document 
summary, modifying a feature includes modifying a feature 
within a pre-defined range or percentage, modifying an 
allowed changeable feature that includes modifying a fea- 
ture within a pre-defined range or percentage, determining a 
reliability score that further includes comparing a ranking 
change for an entry by a first editor to a median ranking 
change for the entry by a second editor, determining a 
reliability score that further includes comparing a summary 
change for an entry to a previous summary contained in the 
first results list, determining a reliability score that further 
includes determining a difference in a number or a percent- 
age of click-throughs which result from the editor 
modification, combining the modification made by the editor 
thai further includes multiplying the reliability score by the 
feature modification, using the combined results to produce 
a second results list includes using the combined search 
index by a search engine and wherein the method is per- 
formed at least twice. 

According to a further aspect of this invention a method 
for producing a results list for a search query, the method 
includes producing a first results list of entries from a search 
algorithm, the first results list corresponding to a term in a 
first search query, the first results list being ordered accord- 
ing to relevance rankings, modifying a feature of the first 
results lists by an editor, the modification being made 
according to an editorial rule, determining a reliability score 
for the editor, the reliability score being based upon the 
modification made the editor, combining the modification 
made by the editor in a combined search index, the com- 
bined modification being included in the combined search 
index according to the determined reliability of the editor, 
detecting a difference between the feature contained in the 
first results list and the corresponding modified feature 
contained in the combined search index, and determining a 
rule that corresponds to the detected difference or similarity. 

One or more of the following features may also be 
included: producing a second results list from the search 
algorithm, the second results list corresponding to the term 
in the first search query and modifying the second results 
lists by applying the rule to a feature contained in the second 
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results list, detecting a difference that further includes 
delecting a change to a relevance ranking feature, detecting 
a difference that further includes detecting an added feature, 
detecting a difference that further includes detecting a 
deleted feature and detecting a difference that further 
includes detecting a change to a document summary feature. 

According to a further aspect of this invention a computer 
program product residing on a computer readable medium 
having a plurality of instructions stored thereon which, when 
executed by the processor, cause that processor to produce 
a first results list of entries from a search algorithm, the first 
resultsl list corresponding to a term in a search query, the 
first results list being ordered according to relevance 
rankings, modify a feature of the first lists according to an 
editorial rule, determine a reliability score for the editorial 
modification, the reliability score being based upon the 
editorial modification, combine the editorial modification in 
a combined search index, the combined modification being 
includes in the combined search index according to the 
reliability score of the editor, and use the combined search 
index to produce a second results list which corresponds to 
the term in the search query, the second results lists being 
ordered according to relevance rankings. 

One or more of the following features may also be 
included: wherein the instructions causing a computer to 
modify a feature of the first lists according to an editorial 
rule and further include instructions causing a computer to 
modify a feature according to an editorial rule, wherein the 
instruction causing a computer to modify a feature according 
to an editorial rule and further include instructions causing 
a computer to modify an allowed changeable feature, the 
allowed changeable feature being one or more of relevance 
ranking, entry addition, entry deletion and document 
summary, wherein the instructions causing a computer to 
determine a reliability score for the editorial modification 
and farther include instructions causing a computer to com- 
pare a ranking change for an entry by a first editor to a 
median ranking change for the entry by a second editor. 

Embodiments of the invention may have one or more of 
the following advantages. For example, a combined editorial 
search index is created that automatically assesses the reli- 
ability of each editorial input, which reduces the potential of 
nefarious editorial input. In some embodiments, a combi- 
nation of paid and un-paid editors are used in evaluating 
specific results lists, which further reduces the possibility of 
nefarious inputs from an editor being included in a search 
index. Some embodiments apply machine learning tech- 
niques to the combined editorial feedback search index, i.e., 
detecting a change in relevance rankings to a previously 
generated index for a specific search term, and then deter- 
mining and storing a rule that reflects the detected change. 
The stored rule may then be re-played by a search engine, 
i.e., applying the stored rule to produce a refined results list 
for the specific search term. The stored rules may also be 
re-played by a search engine to re-rank results lists for a 
search term which has not been evaluated by the editorial 
process. 

The details of one or more embodiments of the invention 
are set forth in the accompanying drawings and the descrip- 
tion below. Other features, objects, and advantages of the 
invention will be apparent from the description and 
drawings, and from the claims. 

DESCRIPTION OF DRAWINGS 
FIG. 1 is a diagrammatic view of the Internet; 
FIG. 2 is a diagrammatic view of an editors ranking a 
results list; 
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FIG. 3 is flowchart of a method for creating a combined 
editorial search index; 

FIG. 4A is a flowchart of a machine learning process; 
- FIG. 4B is a flowchart of the method for creating a 
combined editorial search index; 

FIG. 5 is a diagrammatic view of another embodiment of 
a method for creating a combined editorial search index; 
FIG. 6 is a diagrammatic view of another embodiment of 
10 a method for creating a combined editorial search index. 
Like reference symbols in the various drawings indicate 
like elements. 

DETAILED DESCRIPTION 

15 The Internet and the World Wide Web can be viewed as 
a collection of hyperlinked documents with search engines 
as a primary interface for document retrieval. Search engines 
(e.g., lycos.com) allow the user to enter a query and perform 
a search based on that query. A list of potential matches is 

20 then generated that provides links to potentially relevant 
documents. 

Referring to FIG. 1, there is shown a number of users 10 
accessing the Internet via a network 12 that is connected to 
Internet server 14. The Internet server 14 serves web pages 

25 and Internet- based documents 16 to user 10. Internet server 
14 typically incorporates some form of database 18 to store 
and serve documents 16. 

When user 10 wishes to search for information on a 

3Q specific topic, user 10 utilizes search engine 20 running on 
search engine server 22. User 10 enters a search query 24 
into search engine 20, which provides a results list 26 of 
potential sources for information related to the search terms 
of query 24. Each entry 28 on results list 26 is a document 

}5 summary for a specific relevant document (i.e., a web-page) 
16 on the Internet. These documents 16 may be located on 
search engine server 22, Internet server 14, or any other 
server (not shown) on the network 12; and each document is 
associated with a uniform resource locator (URL) address. 

w Typically, each entry 28 includes a hyperlink, which," when 
selected causes a display of Lhc document associated with 
the URL. 

When search engine 20 provides list 26 to user 10 in 
response to query 24, the individual entries in list 26 arc 

j 5 arranged in accordance with their perceived level of rel- 
evance. Search engine 20 may determine the ranking of the 
entries 28 on list 26 by examining the documents themselves 
to determine certain factors, for example, the number of 
documents linked to the entry, or the presence of the search 

50 term within the document itself; etc. The ranking of the 
entries 28 on list 26 may also be determined from a 
previously stored search index of rankings in a database 40. 
As discussed above, a search index may be created by a 
search engine algorithm or by storing editor-created rank- 

55 in S s - 

Search engines can further enhance their document rank- 
ing accuracy by comparing stored queries (qucry-to-qucry 
relationships) to make suggestions to the user concerning 
modifications or supplemental search terms that would bet- 

60 ter tailor the user's query to the specific information they arc 
searching for. Depending on which selection the user makes, 
the user's search will be appended and tailored accordingly. 

Referring to FIG. 2, there is shown a number of editors 60 
who are evaluating and ranking a results list 50 received 

65 from search engine server 22. Results list 50 includes 
multiple entries 52 which include relevance ranks, Rankl-n, 
which correspond to the perceived relevance of an entry. As 
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discussed above, the editorial changes to results list 50 arc words included in the new summary do not match the old 

returned to the search engine server 22 for inclusion in an summary (or in the document itself) then the reliability score 

editorially ranked search index (not shown) that is stored in of the editor would be lowered; or, 

database 40. ^ . . r 

Comparing how far up or down an entry is moved within 

Referring to FIG. 3, there is shown a number of editors, 5 results list 50 by an editor to the median movement made to 

Editorl-EditorN, participating in an editorial feedback pro- that entry by other editors, and if the movement is greater or 

cess 300. In operation, search engine 20 produces an initial lower than the median by a pre-defined limit, then the 

results list 50 that includes relevance rankings for each entry reliability score of the editor would be lowered- 'or 

in the list. Each editor, Editorl-EditorN, may then perform Determining the 'click-through' increase or decrease as a 

SSSnfnT v u 5 h° a T dl i g 10 SCl ° f ^Itofan editorial change in the results list (i.e., if Internet 

%Z iini PP I' 0 "*. , , T 1C Pr ° CCSSCS uscrs acluaJ selcclions of lhc ^ included in a modified 

tlON ; reS P CCllVC, y- Ed t ll0 t na ^ ™ c P~«sscs entry increasc or dccrease by a pre . dc fined limit, then the 

™ u , ■ P k T fi l °, C u ntr0l , W u 1Ch fe u lUrCS ° f thC rcliabilil y score of the editor can also be changed.) 

results list may be modified by and how they may be n . ,■ . . 

modified by each editor. For example, the editorial rule 15 PIcasc reahzc ;. that an y heunslic method could be used to 

process 310A-310N may allow a change in the following ratC Cdll0nal reIiablllt y- 

features: relevance rankings (i.e., changing the position of an Combined editorial search index 360 may be used directly 

entry within the results list), a deletion of an entry, an by search engine to produce other results lists 50 corre- 

addilion of an entry or a modification to a document sum- sponding to the rankings stored in index 360. 

mary. The editorial rule processes 310A-310N may allow 20 Referring now to FIGS. 3 and 4A, there is shown machine 

complete editorial control or may only allow for partial Earning process 370 which may be included in process 300. 

editorial control of the features contained in results list 50. Machine learning process 370 includes a detection process 

For instance, if partial editorial control is implemented: 372 lnal compares/contrasts the similarities/differences 

An editor may be able to change rankings of an entry, but, between the initial results list 50 and the combined editorial 

only within a pre-defined limit, that is only allowing an 25 search indcx 360 Detected changes or similarities from 

editor to move an entry up or down in the rankings by a diction process 372 are then used by a determine rule 

pre-defined number or process 374, to determine a rule related to the detected 

An editor may not be able to move an entry which is ' hangC °^ simi1 ^ The determined rule is then stored 376 

ranked at a pre-defined percentage above or below another , n r< f by *T C " g ™ ™ T^, 3 rcfined 

cnlry; ori 30 results lists 50A which is ranked according to the stored 

A ' ' . „ , , , rules 380. Detection process 372 may include detecting that: 

An editor may not be allowed to add an entry to the results an pntrx , hQC u ^ n a»\ . i r , r cn 

list unless the ranking associated with the new entry is at a *" enlr > hasbecn ***** from r ^ults list 50; or, 

certain pre-defined level; or, an entry has bcen addcd 10 combined search indcx 360; or, 

An editor may not be allowed to change document 35 m ° mry haS bCC " l0Wered in ranking; or ' 

summaries; or, m entry has been raised in ranking; or, 

An editor may only be allowed to make partial changes to a summar >' has been changed, 

a document summary. Machine learning process then determines 374 a rule to 

Please realize that the editorial rule process 310 applies a ?J1 C ? the dcUxlcd chan S e in rankings, and stores that rule 

set of rules to control which features of a results list may be 40 376 for USe by Search engine 20 For example, for a search 

modified and how they may be modified. ! crm: " Micnael Douglas", the combined editorial search 

t, ... . . , , , r „ index rcfiects a change in ranking position from position 10 

The editorial changes made to the features of the resu ls tn nnQitinn a fnr n L limpnt jz t - , ^ 1 . 

lid ;n k, ^;.n W nftA ham i ■ J C _, , 10 position 4 tor a document related to Catherine Zcta- 

s ?S0A T<Z 3 r 10A - 31 °^ ™ t0 K rC t d m ™ dlficd ™ults Jones- (with a URL of 117). Machine learning process 370 

lists 350A-350N respectively Reliability process 320 then slores 376 a ^ mal „ URL m „ " 

applies a set of rules to determine a reliability score for each nn.iiinn Ifi tn nn.i.inn d r nr ,hV u . u , 

rrlitnr RfiinhiiHiF nrn^cr tin t u a . - a position 1U to position 4 lor the search term Michac 

editor Reliability process 320 then uses the determined Dou gl as ". If a new results list is produced which includes an 

reliability score for each editor together with the changes Pnt * mrrMnnrlriino m npi nt ... \ ™ 

c1nf . nr i : a;k~a u i- . -jcaa ^«v, . , , entry corresponding to UKL 117 at position 10 (or some- 

KrlniT?™ ,h I t 350t l ,0 , WeigtU lhC whcrc ncar P° silion «». < he ™* engine can apply 

Swr^JSi « T L CaC r l d T im ° 3 » lhc slored ™ lc which w «' "use the entry dni 

combined cd.lonaearch mdex 360. The rel.ab.luy process spondi l0 rjRL 117 lo be moved to position 4 wUhin the 

loiHin° 0 V .f / C °" r °i. l ° ,' he ??T S 3 ° 0, lh3 i iS ' ncw rcs " lls »««• Plca " nalize that the stored rule is gencr- 

cTlnl < i Tt IZ'f b H edl H l ° nal Chan8 ^' ° r "^T' a "y broadened 10 ca P lure lhc ^ fe learned change, 

°r ' W • ™ a " Cm i ,led 10 ? lakC i e - lhe storcd ^ a "ovc may promote URL 17 to position 

opcranon .r the ed.tor ma.nta.ns a h.gh rcl.ab.l.ty raung jx promoling URL , 17 l0 kjon J 4 from P a ilion ^ 

then the changes made by that editor are given more weight ranging from 5 lo 15 'rum a positions 

in the combined editorial search index 360. For example, „ . ,. 

reliability process 320 may include determining a reliability RcSU '? ,,MS i nay , b f pr ° duCCd f r ° m 3 S,ngle scarch lerm ' 
score by 'or may be produced from several related terms, i.e., terms 

' . . . «) that produce a results list that includes a large number of 

rh™J^2? 8 h a ra ,f mg H change b J cd,lor '° lhc common entries. By grouping results lists fnto common 

diZn m * ^ r u ° rS l ° lhC , Sa T ? n ? m& ' ^ ' f ** cau *° ries lhc numbcr of ^ reduced and the 

dilTcrence .s greater than a pre-defined l.m.t, then the numbcr of edilors rcquired m cvalua(c the , fa 

rel abihty score of the editor would be lowered; or, reduced . Therefore, a group of editor may be assigned to a 

(If a document summary may be changed by an editor) 65 specific search term, or a category of related terms, so the 

comparing words in the new summary to words in the old edilors may be registered to that specific term or category 

summary and if a pre-defined percentage (or numbcr) of and receive notification when new documents arc available 
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and require editorial review. Similarly, the machine learning 
techniques may be applied to learn and store rules based on 
a common category of queries, i.e., the search term "Michael 
Douglas" and all search terms considered common to this 
search term may have a rule applied to a defined category. 5 

Often a small number of search terms are common to a 
relatively large number of search queries, that is, many users 
are searching for the most popular related terms. Therefore, 
for those highly popular terms, the editorial group assigned 
may include paid editors to augment and supervise the '0 
volunteer editorial process. This would allow for a more 
reliable review of the editorial changes before they arc 
combined and provide for more control of the combined 
search index. 

Now referring to FiG. 4B, there is shown a method 400 15 
for creating a combined editorial search index, and, 
optionally, applying machine learning techniques to the 
search index to determine a machine learning rule. Method 
400 includes: producing 410 a results list for a search term 
(or related search terms) by applying a search algorithm to 20 
a set of documents, changing 420, by an editor, a results list 
feature according to a set of editorial rules, determining 430 
the predicted reliability of the changed feature and/or the 
predicted reliability of the editor and combining 440 the 
weighted editorial changes into a combined editorial search 25 
index. Method 400 may also include applying 370 machine 
learning techniques to determine and store a rule based on a 
detected change between the initial results lists and the 
combined search index. 

Now referring to FIG. 5, there is shown a computer 30 
program product 550 residing on a computer readable 
medium 552 having a plurality of instructions 554 stored 
thereon. When executed by processor 556, instructions 554 
cause processor 556 to: produce 540 a results list for a search 
term (or related search terms), input 542 editorial changes of 
features in the results list by applying a set of editorial rules, 
determine 544 the reliability score for an editor and combine 
546 weighted editorial changes into a combined search 
index. Computer program product 550 may also include 
instructions 554 causing processor to: apply 548 machine 
learning techniques to determine and store a rule corre- 
sponding to a detected change between the results list and 
the combined search index. 

Typical embodiments of computer readable medium 552 45 
arc: hard drive 564; tape drive 566; optical drive 568; RAID 
array 570; random access memory 572; and read only 
memory 574. 

Now referring to FIG. 6, there is shown a processor 600 
and memory 602 configured to: produce 540 a results list for 50 
a search term (or related search terms), input 542 editorial 
changes of features in the results list by applying a set of 
editorial rules, determine 544 the reliability score for an 
editor and combine 546 weighted editorial changes into a 
combined search index. Processor 600 and memory 602 may 55 
also be configured to: apply 548 machine learning tech- 
niques to determine and store a rule corresponding to a 
delected change between the results list and the combined 
editorial search index. 

Processor 600 and memory 602 may be incorporated into 60 
a personal computer 610, a network server 612, or a single 
board computer 614. 

A number of embodiments of the invention have been 
described. Nevertheless, it will be understood that various 
modifications may be made without departing from the spirit 65 
and scope of the invention. Accordingly, other embodiments 
arc within the scope of the following claims. 
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What is claimed is: 

1. A computer implemented method for producing a 
results list for a search query, the method comprising: 

producing a first results list of entries from a search 
algorithm, the -first results list corresponding to a term 
in the search query, the first results list being ordered 
according to relevance rankings; 

modifying a feature of the first results lists by an editor, 
the modification being made according to an editorial 
rule; 

determining a reliability score for the editor, the reliability 
score being based upon the modification made the 
editor; 

combining the modification made by the editor in a 
combined search index, the combined modification 
being included in the combined search index according 
to the determined reliability of the editor; and 

using the combined search index to produce a second 
results list which corresponds to the term in the search 
query. 

2. The method of claim 1 wherein producing a first results 
list includes producing a first results list that corresponds to 
at least one search term. 

3. The method of claim 1 wherein modifying a feature 
according to an editorial rule includes modifying any feature 
of the first results list. 

4. The method of claim 1 wherein modifying a feature 
according to an editorial rule includes modifying an allowed 
changeable feature, the allowed changeable feature being 
one or more of relevance ranking, entry addition, entry 
deletion and document summary. 

5. The method of claim 3 wherein modifying a feature 
includes modifying a feature within a pre-defined range or 
percentage. 

6. The method of claim 4 wherein modifying an allowed 
changeable feature comprises modifying a feature within a 
pre-defined range or percentage. 

7. The method of claim 1 wherein determining a reliabil- 
ity score further comprises: 

comparing a ranking change for an entry by a first editor 
to a median ranking change for the entry by a second 
editor. 

8. The method of claim 1 wherein determining a reliabil- 
ity score further comprises: 

comparing a summary change for an entry to a previous 
summary contained in the first results list. 

9. The method of claim 1 wherein determining a reliabil- 
ity score further comprises: 

determining a difference in a number or a percentage of 
click-throughs which result from the editor modifica- 
tion. 

10. The method of claim 1 wherein combining the modi- 
fication made by the editor further comprises: 

multiplying the reliability score by the feature modifica- 
tion. 

11. The method of claim 1 wherein using the combined 
results to produce a second results list includes using the 
combined search index by a search engine. 

12. The method of claim 1 wherein said method is 
performed at least twice. 

13. A computer implemented method for producing a 
results list for a search query, the method comprising: 

producing a first results list of entries from a search 
algorithm, the first results list corresponding to a term 
in a first search query, the first results list being ordered 
according to relevance rankings; 

modifying a feature of the first results lists by an editor, 
the modification being made according to an editorial 
rule; 
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determining a reliability score for the editor, the reliability 
score being based upon the modification made the 
editor; 

combining the modification made by the editor in a 
combined search index, the combined modification 5 
being included in the combined search index according 
to the determined reliability of the editor; 

detecting a difference between the feature contained in the 
first results list and the corresponding modified feature 
contained in the combined search index; and 10 

determining a rule that corresponds to the detected dif- 
ference or similarity. 

14. The method of claim 13 further comprises: 

producing a second results list from the search algorithm, 
the second results list corresponding to the term in the '5 
first search query; 

modifying the second results lists by applying the rule to 
a feature contained in the second results list. 

15. The method of claim 14 wherein detecting a difference 
further comprises: 20 

detecting a change to a relevance ranking feature. 

16. The method of claim 14 wherein detecting a difference 
further comprises: 

detecting an added feature. 25 

17. The method of claim 14 wherein detecting a difference 
further comprises: 

detecting a deleted feature. 

18. The method of claim 14 wherein detecting a difference 
further comprises: 30 

detecting a change to a document summary feature. 

19. A computer program product residing on a computer 
readable medium having a plurality of instructions stored 
thereon which, when executed by the processor, cause that 
processor to: 35 

produce a first results list of entries from a search 
algorithm, the first resultst list corresponding to a term 



,609 B2 

10 

in a search query, the first results list being ordered 
according to relevance rankings; 
modify a feature of the first lists according to an editorial 
rule; 

determine a reliability score for the editorial modification, 
the reliability score being based upon the editorial 
modification; , iJy 

combine the editorial modification in a combined search 
index, the combined modification being ineJuiics in the 
combined search index according to the reliability 
score of the editor; and 

use the combined search index to produce a second results 
list which corresponds to the term in the search query. 

20. The computer program product of claim 19 wherein 
the instructions causing a computer to modify a feature of 
the first lists according to an editorial rule further comprise 
instructions causing a computer to: 

modify a feature according to an editorial rule. 

21. The computer program product of claim 20 wherein 
the instruction causing a computer to modify a feature 
according to an editorial rule further comprise instructions 
causing a computer to: 

modify an allowed changeable feature, the allowed 
changeable feature being one or more of relevance 
ranking, entry addition, entry deletion and document 
summary. 

22. The computer program product of claim 20 wherein 
the instructions causing a computer to determine a reliability 
score for the editorial modification further comprise instruc- 
tions causing a computer to: 

compare a ranking change for an entry by a first editor to 
a median ranking change for the entry by a second 
editor. 
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